Repository navigation
Add external body storage for audit on sql - #6029
Conversation
…into audit-external-bodies
|
So this would be a new feature? |
|
So |
|
|
||
| The unit differs by provider, so the prefix is part of `IAuditPartitionManager`: | ||
|
|
||
| - SQL Server deletes rows one hour at a time. The prefix is the hour, `yyyyMMddHH-`. |
There was a problem hiding this comment.
Is that internval configurable? Will it to chunks? Is that a native SQL server feature or an application query?
There was a problem hiding this comment.
This is goverened by the retention process, in PostgreSQL it drops partitions and in SQL server it deletes by hour. That same process first calls the body storage to delete that time range.
The background process doesn't actually run on that interval however, that's just the granularity it uses to delete the rows.
| - SQL Server deletes rows one hour at a time. The prefix is the hour, `yyyyMMddHH-`. | ||
| - PostgreSQL drops whole day partitions without scanning rows. The prefix is the day, `yyyyMMdd`, and it matches every hour of that day. A day is only dropped once all of it is older than the retention period, so no body is removed earlier than its row would be. | ||
|
|
||
| Prefixes must not be empty and must not contain wildcard or path characters. SQL Server only visits hours that have rows, so an unreferenced body in an hour with no rows stays until that location is cleaned manually. PostgreSQL visits every expired day partition, including pre-created partitions without rows, so it also cleans unreferenced bodies of those days. |
There was a problem hiding this comment.
How to the other implementation deal with retention?
There was a problem hiding this comment.
When you say "the other implementation" do you mean error instance or raven?
Error instance uses a similar background job to delete expired archived rows, and it makes sure the body is removed before doing so. There's no way for it to delete "blocks" of data like audit does because the resolution of errors is not a time-based process and relies on user intervention.
|
|
||
| An invalid value, or missing settings for the chosen type, still fails at startup. Setup creates the directory, container, or bucket. | ||
|
|
||
| Audit settings use the `ServiceControl.Audit/` prefix in the application settings file. Provider-specific validation matches the Error instance, whose settings use the `ServiceControl/` prefix. |
There was a problem hiding this comment.
Is it a conscious decision that primary and audit do not share the same body store?
There was a problem hiding this comment.
Yes, the problem with sharing the large message storage is that you can no longer clean up unused bodies without cooperation between the instances that have different retention windows.
The implementation allows them to share the same storage account, but the bodies themselves are not shared.
|
@rbev the delimiter for the prefix needs to be set by the implementation, example for Azure Blobs, we want the prefix to be Have a look at my implementation of this in https://github.com/Particular/ServiceControl/pull/5318/changes#diff-985fb65dab014cdcb60c6d03ef95370d1059107f6887c478fdd2c175f9decab1 and https://github.com/Particular/ServiceControl/pull/5318/changes#diff-44f054b50a09b2ab13d8985465c7db9b57fd103451e40fc3030ba889c28624e7R24. |
rbev
changed the title
7d3f48d to
4f1dc57
Compare
Problem
Audit's SQL Server and PostgreSQL persisters discard oversized and binary message bodies. Oversized text retains a searchable prefix, but the body API cannot return the complete body. Error's SQL persisters already support external storage for these bodies.
Change
Reuse Error's FileSystem, Azure Blob, and S3 providers for Audit by extracting them into
ServiceControl.Persistence.BodyStorage.RavenDB behavior is unchanged. Existing
TooLargeandNotTextrows retain their previous API responses; discarded bodies cannot be recovered. The new external-body state uses the existing integer column, so no schema migration is needed.Required configuration
External storage is required for Audit SQL Server and PostgreSQL instances. Existing instances must add
ServiceControl.Audit/MessageBody/StorageTypeand the selected provider's settings before restarting after an upgrade.MaxBodySizeToStorebecomes the inline threshold, not a limit on the complete stored body. Azure defaults to theaudit-bodiescontainer; S3 defaults to theaudit-bodies/key prefix.Each instance must use a separate directory, container, or key prefix. Sharing a location can let one instance's retention process delete another instance's bodies.
Design constraints
Sharing the providers avoids duplicating their storage formats, compression, authentication, and stream handling. Error retains its existing configuration and behavior.
Audit retention deletes by hourly prefixes on SQL Server and daily prefixes on PostgreSQL. This preserves PostgreSQL's partition-drop path without scanning message rows. SQL Server only visits hours containing rows, so bodies orphaned by failed inserts into otherwise empty hours can require manual cleanup.
The current design and configuration are documented in Message body storage.